BBN ATIS System Progress Report - June 1990

This paper reports recent progress on the development of the Delphi natural language component of the BBN spoken language system for the ATIS domain, focussing on the comparative evaluation performed by NIST in Jtme, 1990. 1. Delphi and Parlance/Learner Delphi is BBN's research NL system (formerly called CFG), which is based on a unification grarnmax and which incorporates semantics into the unification framework. Delphi is the NL component of the BBN HARC System. Delphi can be changed very quickly, but has no easy way to build knowledge bases rapidly. The Parlance TM system is a commercial NL interface to relational databases, and is based on an ATN parser and grammar. Parlance has the advantage of an extensive knowledge acquisition system called the Learner TM, so our previously reported approach (Ingria and Ramshaw 1989) has been to use the Learner to create a lexicon with morphological and syntactic information, a domain model with semantic information, and mapping rules from the domain model to the database. These knowledge bases were then imported for use by Delphi. As a side-effect of using the Learner, a Parlance configuration for the ATIS domain was created, so results using that system are reported here for comparison. 2. Analysis of Delphi's Performance on the Blind Test The original score of the BBN Delphi system on the 93 sentence ATIS blind test was 53 sentences correct, 2 sentences not correct and 38 sentences not answered. Of the two sentences judged not correct, one is the result of a mistake in the canonical answer set provided by HIST. The other resulted from a lack of agreement on which table a fare is computed from. (There are a few cases in the ATIS database where information can be retrieved by several different paths, sometimes resulting in different data.) These reulsts indicate that the production of incorrect answers is not a significant problem for Delphi, so the remainder of this section focusses on the sentences not answered. Several causes account for the 38 sentences not answered, with the primm-y one being words of word senses that were not seen in the training material, or which simply did get entered into the lexicon. Another important cause was the failure on the part of the system to infer a relationship between known word senses that a human user would have recognized as implicit. A quantitative breakdown is given in Table 1, and more detailed discussion follows. Reason Word senses missing: Lack of inference: Grammar: SNOR bugs: Miscellaneous: #